Skip to content

HADOOP-19386. Avoid adding asynchronous calls indefinitely which can cause Out-of-Memory.#7287

Closed
hfutatzhanghb wants to merge 2 commits intoapache:trunkfrom
hfutatzhanghb:HADOOP-19386-new
Closed

HADOOP-19386. Avoid adding asynchronous calls indefinitely which can cause Out-of-Memory.#7287
hfutatzhanghb wants to merge 2 commits intoapache:trunkfrom
hfutatzhanghb:HADOOP-19386-new

Conversation

@hfutatzhanghb
Copy link
Member

@hfutatzhanghb hfutatzhanghb commented Jan 14, 2025 •
edited
Loading

Description of PR

Currently, checkAsyncCall is invoked after Client#getConnection which has already put the Rpc call object into
Client#Connection#calls. If checkAsyncCall throws exception, there will no places to remove the useless calls until connection is closed.

@hfutatzhanghb hfutatzhanghb mentioned this pull request Jan 14, 2025
Closed
@hadoop-yetus
Copy link

hadoop-yetus commented Jan 14, 2025

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 12m 11s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 💚 mvninstall 35m 13s trunk passed
+1 💚 compile 16m 26s trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 compile 14m 50s trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 checkstyle 1m 19s trunk passed
+1 💚 mvnsite 1m 35s trunk passed
+1 💚 javadoc 1m 16s trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 54s trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 2m 32s trunk passed
+1 💚 shadedclient 34m 44s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 52s the patch passed
+1 💚 compile 15m 40s the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javac 15m 40s the patch passed
+1 💚 compile 14m 53s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 javac 14m 53s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 1m 14s the patch passed
+1 💚 mvnsite 1m 34s the patch passed
+1 💚 javadoc 1m 12s the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 53s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 2m 35s the patch passed
+1 💚 shadedclient 34m 31s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 19m 48s hadoop-common in the patch passed.
+1 💚 asflicense 1m 6s The patch does not generate ASF License warnings.
216m 12s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7287/1/artifact/out/Dockerfile
GITHUB PR #7287
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux c9c726d67b8c 5.15.0-126-generic #136-Ubuntu SMP Wed Nov 6 10:38:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 17172e2
Default Java Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7287/1/testReport/
Max. process+thread count 2155 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7287/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented Jan 15, 2025

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 31s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 💚 mvninstall 33m 55s trunk passed
+1 💚 compile 16m 55s trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 compile 14m 56s trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 checkstyle 1m 17s trunk passed
+1 💚 mvnsite 1m 37s trunk passed
+1 💚 javadoc 1m 14s trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 53s trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 2m 32s trunk passed
+1 💚 shadedclient 34m 33s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 53s the patch passed
+1 💚 compile 15m 42s the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javac 15m 42s the patch passed
+1 💚 compile 14m 51s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 javac 14m 51s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 1m 13s the patch passed
+1 💚 mvnsite 1m 34s the patch passed
+1 💚 javadoc 1m 11s the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 54s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 2m 42s the patch passed
+1 💚 shadedclient 35m 11s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 19m 46s hadoop-common in the patch passed.
+1 💚 asflicense 1m 4s The patch does not generate ASF License warnings.
204m 11s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7287/2/artifact/out/Dockerfile
GITHUB PR #7287
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 2a677a6d2568 5.15.0-126-generic #136-Ubuntu SMP Wed Nov 6 10:38:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / f811cfc
Default Java Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7287/2/testReport/
Max. process+thread count 3154 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7287/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@KeeProMise
Copy link
Member

KeeProMise commented Jan 17, 2025 •
edited
Loading

image
image
Hi, @hfutatzhanghb, I've thought about it, and here's a better approach:

  • Firstly, the router manages its own code.
  • The existing RouterRpcFairnessPolicyController can be utilized.
  • Most crucially, when receiving a response from nn, the async responder may need to retry. At this time, the async responder will attempt to re-add the request to calls. If calls is full, it cannot be added, causing the async responder to hang. Even though when retrying, the async responder can hand over the request to the async handler for retry, the request still cannot be retried when calls is full. Retried requests should have a higher priority than the latest incoming requests because they were sent to the router earlier.

@hfutatzhanghb hfutatzhanghb deleted the HADOOP-19386-new branch February 18, 2025 08:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@anujmodi2021 anujmodi2021 anujmodi2021 left review comments

Assignees

No one assigned

Labels

Common trunk

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@hfutatzhanghb @hadoop-yetus @KeeProMise @anujmodi2021